feat: prefix dictionaries (refPrefix) for delta compression#27
Merged
Conversation
Bind ZSTD_CCtx_refPrefix / ZSTD_DCtx_refPrefix as ZstdCompressCtx.refPrefix(MemorySegment) / ZstdDecompressCtx.refPrefix(...): a single-use, by-reference raw-content dictionary for the next frame only — the building block for delta compression (compress a new version against a similar previous one). No digest, no copy, no dictionary ID written; the decompressor must reference the same prefix to decode. Segment-only by design. refPrefix references the prefix by pointer (unlike loadDictionary, which copies internally), so the prefix must outlive the next compression — a contract only a caller-owned native segment can honour. A byte[] overload would have to copy, defeating the zero-copy point and forcing a context-held arena to keep the copy alive; heap callers should use the copying loadDictionary instead. RefPrefixTest proves the prefix is actually applied (16 KiB of random data, identical to the prefix, compresses to tens of bytes at level 19 only with the prefix) and is required to decode — a round-trip-only test would pass even if refPrefix were a no-op. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- prove refPrefix is single-use: prefix set once, two compressions, only the first frame shrinks (random data at level 19); second frame decodes with a plain decoder, confirming it does not stick across frames. - assert ZstdDecompressCtx.refPrefix also rejects a heap segment (was only covered on the compress side). - import java.util.Arrays instead of the fully-qualified inline reference. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Binds
ZSTD_CCtx_refPrefix/ZSTD_DCtx_refPrefixasZstdCompressCtx.refPrefix(MemorySegment)/ZstdDecompressCtx.refPrefix(MemorySegment):a single-use, by-reference raw-content prefix applied to the next frame only —
the building block for delta compression (compress a new version against a
similar previous one, storing little more than the difference).
Unlike
loadDictionary, a prefix is not digested, not copied, and writes nodictionary ID; the decompressor must reference the same prefix to decode.
Why segment-only (no
byte[]overload)refPrefixreferences the prefix by pointer — it must outlive the nextcompression. A
byte[]overload would have to copy, which both defeats thezero-copy point and forces a context-held arena to keep the copy alive until
consumed (extra lifecycle state in
reset/close). Heap callers that want acopy already have the right tool:
loadDictionary. So the API is segment-only,caller-owned lifetime — matching zstd's semantics exactly.
Tests
RefPrefixTest(4 cases): round-trip with matching prefix, null clears, heapsegment rejected, and the key one — prefix is applied and required to decode:
16 KiB of random data identical to the prefix compresses to tens of bytes at
level 19 only with the prefix, and the frame cannot be decoded without it. A
round-trip-only test would pass even if
refPrefixwere a no-op; this one failsif the prefix isn't actually wired through (it caught a real by-reference
lifetime bug during development).
Full module suite: 200 tests green, checkstyle clean.
Notes
docs/supported.mdupdated (Advanced parameters 12 → 14/38; zstd-jni does notexpose
refPrefix).CHANGELOG.mdUnreleased entry added.🤖 Generated with Claude Code